Config reference documentation autogeneration #2033

ilyakuz-db · 2024-12-18T18:24:19Z

Changes

Documentation autogeneration tool. This tool uses same annotations_*.yml files as in json-schema

Result will go there and there

Tests

Manually

eng-dev-ecosystem-bot · 2024-12-18T20:50:54Z

Test Details: go/deco-tests/12401106836

…erence-doc-autogen

github-actions · 2025-01-03T12:53:29Z

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

PR number: 2033
Commit SHA: c6703c13637c0d85e6754e4c802f38052304a5d6

Checks will be approved automatically on success.

…erence-doc-autogen

pietern

Did you consider checking in the Markdown output as well?

That can serve as both a check that the contents is OK and that it doesn't inadvertently drift when changes to the schema or annotations are made.

bundle/internal/annotation/main.go

bundle/internal/docs/README.md

bundle/internal/docs/markdown.go

pietern · 2025-01-15T11:06:57Z

bundle/internal/docs/nodes.go

+	return isCustomField
+}
+
+func extractNodes(prefix string, props map[string]*jsonschema.Schema, refs map[string]jsonschema.Schema, customFields map[string]bool) []rootProp {


Same as above.

pietern · 2025-01-15T11:07:09Z

bundle/internal/docs/refs.go

+	return true
+}
+
+func resolveAdditionaProperties(v *jsonschema.Schema) *jsonschema.Schema {


bundle/internal/schema/annotations.go

go.mod

pietern · 2025-01-15T11:12:54Z

bundle/internal/docs/nodes.go

+		return nodes[i].Title < nodes[j].Title
+	})
+	return nodes
+}


It's very hard to tell what this function does and how it is supposed to work. Please include a few more comments with how you expect it to work, and if possible even a testcase or two to make sure it does work the way you assume. As is we need to run it and check the output to confirm this.

Added few test cases and re-arranged the code with more comments to highlight important parts

ilyakuz-db · 2025-01-15T15:22:54Z

Did you consider checking in the Markdown output as well?

That can serve as both a check that the contents is OK and that it doesn't inadvertently drift when changes to the schema or annotations are made.

You mean to remove from .gitignore, right?

denik · 2025-01-16T08:56:05Z

Makefile

 schema:
 	go run ./bundle/internal/schema ./bundle/internal/schema ./bundle/schema/jsonschema.json

+docs: vendor


docs: vendor

We don't add this implicitly anymore to other targets. If you're on CI, you can always specify both with "make vendor docs".

BTW, do you intended to run it on CI?

BTW, do you intended to run it on CI?

Currently the only use case of using that tool is to checkout CLI repo and run locally

We don't add this implicitly anymore to other targets

By implicitly you mean using make deps or in general?

I mean the make entry should be:

docs: ...

no dependency on vendor

Could you elaborate little bit more on the reasoning behind "We don't add this implicitly anymore to other targets" so I can understand how to better fix it?

Is it just that we don't want to rely on make deps syntax and this is still allowed?

docs: make vendor go run ....

Removed and update the readme

denik · 2025-01-16T08:59:23Z

Makefile

 schema:
 	go run ./bundle/internal/schema ./bundle/internal/schema ./bundle/schema/jsonschema.json

+docs: vendor
+	go run ./bundle/internal/docs ./bundle/internal/schema ./bundle/internal/docs


Why is this part if /bundle/internal/ ?

internal is meant for code that is not meant to be used outside of the package, but this is clearly intended to be used outside the package, that's why it's in Makefile.

How about placing this at top level? /docsgen/ ?

I see you're following existing structure, so it's a more of a question for @pietern

I see you're following existing structure,

Yes I reused same approach as with json schema generation which is used in .codegen.json along with other internal scripts

This is bundle specific, so bundle/docsgen would be more appropriate.

It doesn't need to be internal.

…erence-doc-autogen

ilyakuz-db · 2025-01-20T16:18:25Z

@pietern @denik could you take another look? Want to merge it so docs team could start using this tool

denik · 2025-01-21T13:52:08Z

bundle/internal/annotation/file.go

+)
+
+// Parsed file with annotations, expected format:
+// github.com/databricks/cli/bundle/config.Bundle:


This is copied, right?

It would be much easier to review if PRs was split into

refactoring: copy / renames

new functionality

Yes, this was moved as is from schema package into the new package

It would be much easier to review if PRs was split into

refactoring: copy / renames
new functionality

Agree but it feels too painful to split now. For next PRs I'll try to follow more convenient approach

pietern · 2025-01-27T11:27:00Z

libs/jsonschema/schema.go

+
+	// Examples of the value for properties in the schema.
+	// https://json-schema.org/understanding-json-schema/reference/annotations
+	Examples []any `json:"examples,omitempty"`


Do you know if the schema check we perform also validates that entries here match the containing schema?

I think it only tests yamls against the schema but not the schema consistency itsef

I.e. this is allowed

In general spec says that example values should validate against schema but it isn't used and not required
https://json-schema.org/understanding-json-schema/reference/annotations see draft 6

Worth to mention that this field currently is not used in schema generation

pietern · 2025-01-27T11:33:23Z

Makefile

 schema:
 	go run ./bundle/internal/schema ./bundle/internal/schema ./bundle/schema/jsonschema.json

+docs: vendor
+	go run ./bundle/internal/docs ./bundle/internal/schema ./bundle/internal/docs


This is bundle specific, so bundle/docsgen would be more appropriate.

It doesn't need to be internal.

…erence-doc-autogen

ilyakuz-db added 2 commits December 18, 2024 17:07

tmp: Documentation autogeneration script

8c58d48

fix: Custom table markup changes

e27cc9a

ilyakuz-db temporarily deployed to test-trigger-is December 18, 2024 18:24 — with GitHub Actions Inactive

ilyakuz-db added 2 commits December 18, 2024 19:59

fix: Styling issues

fe4c6b8

fix: Remove openAPI docs

cfa2be3

ilyakuz-db temporarily deployed to test-trigger-is December 18, 2024 19:53 — with GitHub Actions Inactive

ilyakuz-db temporarily deployed to test-trigger-is December 18, 2024 19:54 — with GitHub Actions Inactive

ilyakuz-db added 2 commits December 18, 2024 21:12

fix: Remove extra headings

10e0d27

fix: Small readme

64fa2bf

ilyakuz-db temporarily deployed to test-trigger-is December 18, 2024 20:50 — with GitHub Actions Inactive

ilyakuz-db added 5 commits December 23, 2024 12:25

Merge branch 'main' of github.com:databricks/cli into feat/config-ref…

40c4b3a

…erence-doc-autogen

feat: Add examples to the docs

820dd5f

chore: Extract annotation package

460a455

feat: More explicit type for arrays

541c3e3

feat: Support for resources

2aadfcb

ilyakuz-db requested review from pietern, andrewnester, shreyas-goenka and denik as code owners January 2, 2025 17:24

fix: Updated styles

c6703c1

ilyakuz-db added 6 commits January 7, 2025 16:52

fix: Styling

fe6ba76

fix: Description of root types with additional properties

c355fbf

docs: Add override for volume spec

f9278c2

fix: Missing array types

6c5268a

fix: Sync annotations

bad77bd

Merge branch 'main' of github.com:databricks/cli into feat/config-ref…

954ef76

…erence-doc-autogen

pietern reviewed Jan 15, 2025

View reviewed changes

fix: Add vendor step to make docs

14b2c86

ilyakuz-db added 9 commits January 15, 2025 16:23

fix: Stage output changes

c74ab75

fix: Split annotations/main.go in 2 files

1b73105

fix: Move templates to separate files

e5497f7

fix: Typo

33bd3b8

fix: Remove extra dependency

b91bb4d

fix: More documentation for internal types

14721de

fix: Remove type inconsistency in maps

26466d2

fix: Rename fields to make it more clear

d27282c

fix: More ducmentation + tests

13181fc

denik reviewed Jan 16, 2025

View reviewed changes

ilyakuz-db requested a review from pietern January 16, 2025 11:28

ilyakuz-db added 2 commits January 20, 2025 12:26

Merge branch 'main' of github.com:databricks/cli into feat/config-ref…

f04a841

…erence-doc-autogen

fix: Update artifacts

e40b53f

ilyakuz-db temporarily deployed to test-trigger-is January 20, 2025 11:27 — with GitHub Actions Inactive

ilyakuz-db requested a review from denik January 20, 2025 16:17

denik reviewed Jan 21, 2025

View reviewed changes

fix: Remove unnecessary vendor make dependency

2154f2a

ilyakuz-db temporarily deployed to test-trigger-is January 21, 2025 15:05 — with GitHub Actions Inactive

ilyakuz-db changed the title ~~feat: Config reference documentation autogeneration~~ Config reference documentation autogeneration Jan 21, 2025

pietern approved these changes Jan 27, 2025

View reviewed changes

ilyakuz-db added 3 commits January 27, 2025 16:42

Move to bundle/docsgen

e5e47f4

Merge branch 'main' of github.com:databricks/cli into feat/config-ref…

21bd217

…erence-doc-autogen

Regenerate docs with latest SDK changes

b5fda64

ilyakuz-db temporarily deployed to test-trigger-is January 27, 2025 15:47 — with GitHub Actions Inactive

Update schema

bc88fe3

ilyakuz-db temporarily deployed to test-trigger-is January 27, 2025 15:49 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Config reference documentation autogeneration #2033

Config reference documentation autogeneration #2033

ilyakuz-db commented Dec 18, 2024 •

edited

Loading

eng-dev-ecosystem-bot commented Dec 18, 2024

github-actions bot commented Jan 3, 2025

pietern left a comment

pietern Jan 15, 2025

pietern Jan 15, 2025

pietern Jan 15, 2025

ilyakuz-db Jan 16, 2025

ilyakuz-db commented Jan 15, 2025 •

edited

Loading

denik Jan 16, 2025

ilyakuz-db Jan 16, 2025

denik Jan 16, 2025

ilyakuz-db Jan 16, 2025

ilyakuz-db Jan 21, 2025

denik Jan 16, 2025

ilyakuz-db Jan 16, 2025

pietern Jan 27, 2025

ilyakuz-db commented Jan 20, 2025

denik Jan 21, 2025

ilyakuz-db Jan 21, 2025

ilyakuz-db Jan 21, 2025

pietern Jan 27, 2025

ilyakuz-db Jan 27, 2025

pietern Jan 27, 2025

Config reference documentation autogeneration #2033

Are you sure you want to change the base?

Config reference documentation autogeneration #2033

Conversation

ilyakuz-db commented Dec 18, 2024 • edited Loading

Changes

Tests

eng-dev-ecosystem-bot commented Dec 18, 2024

github-actions bot commented Jan 3, 2025

pietern left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilyakuz-db commented Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilyakuz-db commented Jan 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilyakuz-db commented Dec 18, 2024 •

edited

Loading

ilyakuz-db commented Jan 15, 2025 •

edited

Loading